As digital image data are widely used, techniques for processing image data are widely used.
For example, an image processing device using image data changes the resolution of image data as necessary (for example, see PTL 1).
On the other hand, a remote monitoring system using multiple image-capturing devices and an image processing device processing images is widely employed.
There is a case in which image data captured with the remote monitoring system described above has a resolution lower than a required resolution by performance of the image-capturing device or external factors.
A super resolution technique is one of techniques for processing the resolution of such image data, and more particularly, the super resolution technique is a technique for enhancing the resolution of the image data.
The super resolution technique includes the following techniques.
The first super resolution technique is multi-image super resolution technique. The multi-image super resolution technique is a technique for generating a single high resolution image data by using a motion picture or a multiple continuously-shot image data (multiple frames) (for example, see PTL 2). As described above, in order to achieve a high resolution, the multi-image super resolution technique requires image data of multiple images. For this reason, the multi-image super resolution technique is unable to generate high resolution image data from a single image data.
The second super resolution technique is learning-based super resolution technique. The learning-based super resolution technique is a technique for generating a dictionary on the basis of learning in advance, and enhancing the resolution of a single image data by using the dictionary (for example, see PTL 3). Since the learning-based super resolution technique uses the dictionary, the learning-based super resolution technique can achieve a super resolution of which resolution is higher than the multi-image super resolution technique.
An image processing device using learning-based super resolution technique will be further described with reference to drawings.
The image processing device using the learning-based super resolution technique generally includes a learning phase and a super resolution phase.
FIG. 7 is a block diagram illustrating an example of a configuration of an image processing system 900 including an image processing device 910 using the learning-based super resolution technique.
The image processing system 900 includes the image processing device 910, a learning device 920, and a dictionary 930.
The learning device 920 generates data (dictionary data) held in the dictionary 930 by using a learning image 51. Namely, the learning device 920 processes the learning phase.
The learning device 920 and the learning phase will be described with reference to FIGS. 8 and 9.
FIG. 8 is a block diagram illustrating an example of a configuration of the learning device 920.
FIG. 9 is a drawing for describing the learning phase.
The learning device 920 includes a reception unit 921, a blurred image generation unit 922, a patch pair generation unit 923, and a registration unit 924.
The reception unit 921 receives a high resolution image (learning image 51) for learning, and sends the learning image 51 to the blurred image generation unit 922 and the patch pair generation unit 923.
The blurred image generation unit 922 generates a low resolution image (blurred image 52) having a lower resolution on the basis of the learning image 51. As shown in FIG. 9, the blurred image generation unit 922 may generate a plurality of blurred images 52. The blurred image generation unit 922 sends the blurred image 52 to the patch pair generation unit 923.
The patch pair generation unit 923 extracts an image in a predetermined range (high resolution patch 511) from the learning image 51. Then, the patch pair generation unit 923 extracts an image corresponding to the extracted high resolution patch 511 (low resolution patch 521) from the blurred image 52. The patch pair generation unit 923 generates a patch pair 531 obtained by combining the high resolution patch 511 and the low resolution patch 521. The patch pair generation unit 923 sends the patch pair 531 to the registration unit 924.
The registration unit 924 stores the patch pair 531 to the dictionary 930.
The explanation will be made with reference to FIG. 7 again.
As described above, the dictionary 930 stores the patch pair 531.
The image processing device 910 composes a restoration image 55 generated by making the input image 54 into a high resolution by using the patch pair 531 of the dictionary 930. Namely, the image processing device 910 processes the super resolution phase.
The image processing device 910 and the super resolution phase will be described with reference to FIGS. 10 and 11.
FIG. 10 is a block diagram illustrating an example of a configuration of the image processing device 910.
FIG. 11 is a drawing for describing a super resolution phase.
The image processing device 910 includes a patch generation unit 911, a selection unit 912, and a composite unit 913.
The patch generation unit 911 receives the input image 54. Then, the patch generation unit 911 generates a patch (input patch 541) to be compared with the low resolution patch 521 of the patch pair 531 on the basis of the input image 54. The patch generation unit 911 sends the generated input patch 541 to the selection unit 912.
The selection unit 912 selects the patch pair 531 in the dictionary 930 on the basis of the input patch 541. More specifically, the selection unit 912 operates, for example, as described below. The selection unit 912 calculates similarity between the input patch 541 and all the low resolution patches 521 of the patch pairs 531. Then, the selection unit 912 selects a patch pair 531 including the low resolution patch 521 that is the most similar. The high resolution patch 511 of the selected patch pair 531 becomes a patch used for composition (restoration patch 551).
The selection unit 912 selects the patch pairs 531 corresponding to all the input patches 541, and notifies the composite unit 913 of the selected patch pairs 531.
The composite unit 913 composes the restoration image 55 by using the high resolution patch 511 of the notified patch pair 531 as the restoration patch 551.